Multi-center validation of an artificial intelligence system for detection of COVID-19 on chest radiographs in symptomatic patients.

OBJECTIVES: While chest radiograph (CXR) is the first-line imaging investigation in patients with respiratory symptoms, differentiating COVID-19 from other respiratory infections on CXR remains challenging. We developed and validated an AI system for COVID-19 detection on presenting CXR. METHODS: A deep learning model (RadGenX), trained on 168,850 CXRs, was validated on a large international test set of presenting CXRs of symptomatic patients from 9 study sites (US, Italy, and Hong Kong SAR) and 2 public datasets from the US and Europe. Performance was measured by area under the receiver operator characteristic curve (AUC). Bootstrapped simulations were performed to assess performance across a range of potential COVID-19 disease prevalence values (3.33 to 33.3%). Comparison against international radiologists was performed on an independent test set of 852 cases. RESULTS: RadGenX achieved an AUC of 0.89 on 4-fold cross-validation and an AUC of 0.79 (95%CI 0.78-0.80) on an independent test cohort of 5,894 patients. Delong's test showed statistical differences in model performance across patients from different regions (p < 0.01), disease severity (p < 0.001), gender (p < 0.001), and age (p = 0.03). Prevalence simulations showed the negative predictive value increases from 86.1% at 33.3% prevalence, to greater than 98.5% at any prevalence below 4.5%. Compared with radiologists, McNemar's test showed the model has higher sensitivity (p < 0.001) but lower specificity (p < 0.001). CONCLUSION: An AI model that predicts COVID-19 infection on CXR in symptomatic patients was validated on a large international cohort providing valuable context on testing and performance expectations for AI systems that perform COVID-19 prediction on CXR. KEY POINTS: • An AI model developed using CXRs to detect COVID-19 was validated in a large multi-center cohort of 5,894 patients from 9 prospectively recruited sites and 2 public datasets. • Differences in AI model performance were seen across region, disease severity, gender, and age. • Prevalence simulations on the international test set demonstrate the model's NPV is greater than 98.5% at any prevalence below 4.5%.

Machine learning application for the prediction of SARS-CoV-2 infection using blood tests and chest radiograph.

Triaging and prioritising patients for RT-PCR test had been essential in the management of COVID-19 in resource-scarce countries. In this study, we applied machine learning (ML) to the task of detection of SARS-CoV-2 infection using basic laboratory markers. We performed the statistical analysis and trained an ML model on a retrospective cohort of 5148 patients from 24 hospitals in Hong Kong to classify COVID-19 and other aetiology of pneumonia. We validated the model on three temporal validation sets from different waves of infection in Hong Kong. For predicting SARS-CoV-2 infection, the ML model achieved high AUCs and specificity but low sensitivity in all three validation sets (AUC: 89.9-95.8%; Sensitivity: 55.5-77.8%; Specificity: 91.5-98.3%). When used in adjunction with radiologist interpretations of chest radiographs, the sensitivity was over 90% while keeping moderate specificity. Our study showed that machine learning model based on readily available laboratory markers could achieve high accuracy in predicting SARS-CoV-2 infection.

Imaging Profile of the COVID-19 Infection: Radiologic Findings and Literature Review.

PURPOSE: To present the findings of 21 coronavirus disease 2019 (COVID-19) cases from two Chinese centers with CT and chest radiographic findings, as well as follow-up imaging in five cases. MATERIALS AND METHODS: This was a retrospective study in Shenzhen and Hong Kong. Patients with COVID-19 infection were included. A systematic review of the published literature on radiologic features of COVID-19 infection was conducted. RESULTS: The predominant imaging pattern was of ground-glass opacification with occasional consolidation in the peripheries. Pleural effusions and lymphadenopathy were absent in all cases. Patients demonstrated evolution of the ground-glass opacities into consolidation and subsequent resolution of the airspace changes. Ground-glass and consolidative opacities visible on CT are sometimes undetectable on chest radiography, suggesting that CT is a more sensitive imaging modality for investigation. The systematic review identified four other studies confirming the findings of bilateral and peripheral ground glass with or without consolidation as the predominant finding at CT chest examinations. CONCLUSION: Pulmonary manifestation of COVID-19 infection is predominantly characterized by ground-glass opacification with occasional consolidation on CT. Radiographic findings in patients presenting in Shenzhen and Hong Kong are in keeping with four previous publications from other sites.© RSNA, 2020See editorial by Kay and Abbara in this issue.

Detection of COVID-19 Using Deep Learning Algorithms on Chest Radiographs.

PURPOSE: To evaluate the performance of a deep learning (DL) algorithm for the detection of COVID-19 on chest radiographs (CXR). MATERIALS AND METHODS: In this retrospective study, a DL model was trained on 112,120 CXR images with 14 labeled classifiers (ChestX-ray14) and fine-tuned using initial CXR on hospital admission of 509 patients, who had undergone COVID-19 reverse transcriptase-polymerase chain reaction (RT-PCR). The test set consisted of a CXR on presentation of 248 individuals suspected of COVID-19 pneumonia between February 16 and March 3, 2020 from 4 centers (72 RT-PCR positives and 176 RT-PCR negatives). The CXR were independently reviewed by 3 radiologists and using the DL algorithm. Diagnostic performance was compared with radiologists' performance and was assessed by area under the receiver operating characteristics (AUC). RESULTS: The median age of the subjects in the test set was 61 (interquartile range: 39 to 79) years (51% male). The DL algorithm achieved an AUC of 0.81, sensitivity of 0.85, and specificity of 0.72 in detecting COVID-19 using RT-PCR as the reference standard. On subgroup analyses, the model achieved an AUC of 0.79, sensitivity of 0.80, and specificity of 0.74 in detecting COVID-19 in patients presented with fever or respiratory systems and an AUC of 0.87, sensitivity of 0.85, and specificity of 0.81 in distinguishing COVID-19 from other forms of pneumonia. The algorithm significantly outperforms human readers (P<0.001 using DeLong test) with higher sensitivity (P=0.01 using McNemar test). CONCLUSIONS: A DL algorithm (COV19NET) for the detection of COVID-19 on chest radiographs can potentially be an effective tool in triaging patients, particularly in resource-stretched health-care systems.

Development and validation of risk prediction models for COVID-19 positivity in a hospital setting.

OBJECTIVES: To develop: (1) two validated risk prediction models for coronavirus disease-2019 (COVID-19) positivity using readily available parameters in a general hospital setting; (2) nomograms and probabilities to allow clinical utilisation. METHODS: Patients with and without COVID-19 were included from 4 Hong Kong hospitals. The database was randomly split into 2:1: for model development database (n = 895) and validation database (n = 435). Multivariable logistic regression was utilised for model creation and validated with the Hosmer-Lemeshow (H-L) test and calibration plot. Nomograms and probabilities set at 0.1, 0.2, 0.4 and 0.6 were calculated to determine sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). RESULTS: A total of 1330 patients (mean age 58.2 ± 24.5 years; 50.7% males; 296 COVID-19 positive) were recruited. The first prediction model developed had age, total white blood cell count, chest x-ray appearances and contact history as significant predictors (AUC = 0.911 [CI = 0.880-0.941]). The second model developed has the same variables except contact history (AUC = 0.880 [CI = 0.844-0.916]). Both were externally validated on the H-L test (p = 0.781 and 0.155, respectively) and calibration plot. Models were converted to nomograms. Lower probabilities give higher sensitivity and NPV; higher probabilities give higher specificity and PPV. CONCLUSION: Two simple-to-use validated nomograms were developed with excellent AUCs based on readily available parameters and can be considered for clinical utilisation.

Frequency and Distribution of Chest Radiographic Findings in Patients Positive for COVID-19.

Background Current coronavirus disease 2019 (COVID-19) radiologic literature is dominated by CT, and a detailed description of chest radiography appearances in relation to the disease time course is lacking. Purpose To describe the time course and severity of findings of COVID-19 at chest radiography and correlate these with real-time reverse transcription polymerase chain reaction (RT-PCR) testing for severe acute respiratory syndrome coronavirus 2, or SARS-CoV-2, nucleic acid. Materials and Methods This is a retrospective study of patients with COVID-19 confirmed by using RT-PCR and chest radiographic examinations who were admitted across four hospitals and evaluated between January and March 2020. Baseline and serial chest radiographs (n = 255) were reviewed with RT-PCR. Correlation with concurrent CT examinations (n = 28) was performed when available. Two radiologists scored each chest radiograph in consensus for consolidation, ground-glass opacity, location, and pleural fluid. A severity index was determined for each lung. The lung scores were summed to produce the final severity score. Results The study was composed of 64 patients (26 men; mean age, 56 years ± 19 [standard deviation]). Of these, 58 patients had initial positive findings with RT-PCR (91%; 95% confidence interval: 81%, 96%), 44 patients had abnormal findings at baseline chest radiography (69%; 95% confidence interval: 56%, 80%), and 38 patients had initial positive findings with RT-PCR testing and abnormal findings at baseline chest radiography (59%; 95% confidence interval: 46%, 71%). Six patients (9%) showed abnormalities at chest radiography before eventually testing positive for COVID-19 with RT-PCR. Sensitivity of initial RT-PCR (91%; 95% confidence interval: 83%, 97%) was higher than that of baseline chest radiography (69%; 95% confidence interval: 56%, 80%) (P = .009). Radiographic recovery (mean, 6 days ± 5) and virologic recovery (mean, 8 days ± 6) were not significantly different (P = .33). Consolidation was the most common finding (30 of 64; 47%) followed by ground-glass opacities (21 of 64; 33%). Abnormalities at chest radiography had a peripheral distribution (26 of 64; 41%) and lower zone distribution (32 of 64; 50%) with bilateral involvement (32 of 64; 50%). Pleural effusion was uncommon (two of 64; 3%). The severity of findings at chest radiography peaked at 10-12 days from the date of symptom onset. Conclusion Findings at chest radiography in patients with coronavirus disease 2019 frequently showed bilateral lower zone consolidation, which peaked at 10-12 days from symptom onset. © RSNA, 2020.